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Introduction 


On*  of  the  foremost  problems  in  psychologicel  meesurement  of  per¬ 
sonnel  performance  is  the  selection  of  e  reliable,  as  well  as  valid  rating 
method.  An  equally  large  problem,  once  the  method  is  selected,  is  who 
sh  ould  accomplish  the  evaluations. 

The  US  Army  Enlisted  Evaluation  Center  utilises  peer  or  co-worker 
ratings  as  a  criterion  for  validation  of  Evaluation  Tests  (ET)  and 
Commander's  Evaluation  Reports  (CER) .  This  particular  rating  method 
was  selected  in  lieu  of  other  methods  because:  1)  numerous  journal 
articles  and  books  (Hollander,  1954a,  1954b,  1956a,  1956b,  1957;  Doll 
&  Longo,  1962;  Turk,  1961;  Tiffin  &  McCormick,  1960;  Cornbach.  1960; 
Whisler  &  Harper,  1962)  advocate  and  support  the  superiority  of  peer- 
ratings,  particularly  their  reliability  and  validity;  2)  supervisory 
criterion  ratings  would  overlap  with  the  supervisor's  CER  ratings;  and 
3)  the  use  of  peer-rating? at  USAEEC  have  proven  to  be  the  most  re¬ 
liable  as  well  as  practical  criterion. 

In  many  instances,  however,  it  is  not  possible  to  obtain  suffi¬ 
cient  peer-ratings  to  be  used  as  a  criterion.  This  is  particularly 
true  at  the  higher  Military  Occupational  Speciality  (MOS)  skill  levels 
in  which  an  insufficient  number  of  higher  grade  personnel  work  closely 
enough  together  to  be  qualified  to  rate  one  another.  A  possible 
solution  to  this  problem  is  to  use  either  self-ratings  for  validation 
of  the  ET  and  CER  or  supervisory  ratings  as  a  criterion  solely  for  the 
validation  of  the  ET. 

Purpose.  Thus,  it  is  the  objective  of  this  study  to  compare  peer-ratings, 
self-ratings,  and  supervisory  ratings  to  determine  if  the  self  or  super¬ 
visory  ratings  correlate  significantly  to  the  peer-ratings  to  warrant 
their  use  as  a  criterion. 
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Method 

Peer,  self,  and  supervisory  ratings  were  obtained  during  the  period 
12  -  16  April  1963  on  215  enlisted  men  stationed  at  Fort  Riley,  Kansas. 
Semples  were  selected  from  36  separate  MOS  skill  levels. 

Three  peer-ratings  were  rendered  on  each  ratee  by  enlisted  men  with¬ 
in  the  same  MOS  skill  level.  Each  supervisor  accomplishing  the  ratings 
was  the  immediate  superior  of  the  ratee.  All  raters  had  observed  the 
man's  on-the-job  performance  at  least  several  times  a  week  and  had  worked 
with  or  supervised  the  man  a  minimum  of  one  month.  The  purpose  of  the 
ratings  was  explained  to  the  raters  and  the  need  for  rendering  accurate 
ratings  was  emphasised. 

The  ratings  were  accomplished  on  two  factors,  "Overall  Performance" 
and  "Promotion  Qualification,"  using  a  discreet  graphic  scale  ranging 
from  1  (poor)  to  11  (exceptional)  as  shown  on  EPEECE  Form  70  (see  Appendix). 

Pearson  product-moment  correlation  coefficients  were  computed  between 
the  three  types  of  ratings  on  both  factors.  These  correlations  were  then 
tested  for  significant  differences.  The  correlations  were  also  computed 
between  the  two  rating  factors  using  each  of  the  rating  types,  re¬ 
spectively,  to  determine  the  amoun"  of  agreement  between  the  scales. 
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Results 

An  lntercorrelstion  matrix  for  the  three  types  of  ratings  obtained 
on  the  "Overall  Performance"  scale  is  shown  in  Table  l|  The  inter- 
correlation  matrix  for  the  "Promotion  Qualification"  scale  is  presented 
in  Table  II.  The  highest  correlation  coefficients  were  between  the 
peer-ratings  and  the  supervisors'  ratings  (.3922  and  .4766)  on  the  two 
scales,  respectively.  Whereas, the  lowest  correlations  on  both  rating 
scales  were  between  the  self-ratings  and  the  supervisory  ratings  (.2463 
and  .2397).  The  reliability  of  the  peer-ratings  was  .4625,  significant 
at  the  .001  level. 

The  correlation  coefficients  between  the  two  scales  computed  from 
each  type  of  rating  are  shown  in  Table  III.  Each  correlation  coefficient 
was  significantly  different  from  eero  at  the  .001  significance  level. 

The  two  scales  were  most  highly  correlated  (.8494)  using  the  self- 
ratings  and  the  least  correlation  (.7296)  was  found  using  the  pser- 
rat lngs . 
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Table  I 


Intercorrelation  Matrix  Between  Three  Rating 
Methoda  on  Overall  Performance* 


Rating  Method 

Peer  Self  Supervisory 

Peer  --  .2536  .3922 


Rating  Method  Self 

Supervisor 


.  2463 


Table  II 


Intercorrelation  Matrix  Between  Three  Rating 
Methods  on  Promotion  Qualification* 


Rating  Method 
Peer  Self 

Peer  --  .3111 

Rating  Method  Self 

Supervisor 


Supervisory 

.4766 

.2397 


Table  III 

Correlations  Between  Overall  Performance 
Scale  and  Promotion  Qualification  Scale 
from  Three  Rating  Methods* 

Rating  Scales 


OP  vs  PQ** 

Peer  .7296 

Rating  Method  Self  .8494 

Supervisor  .7731 

♦All  correlation  coefficients  are  significant  at  the  .01  level. 
♦♦Overall  Performance  (OP)  vs  Promotion  Qual i f icat inn  (PQ) . 
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Discus* ion 


The  purpose  of  this  study  was  to  investigate  and  compare  peer- 
ratings,  the  present  criterion,  with  self-ratings,  and  supervisory 
ratings.  If  either  the  self  or  supervisory  ratings  were  signifi¬ 
cantly  correlated  with  the  peer-ratings ,  they  could  be  substituted 
as  the  criterion  when  an  insufficient  number  of  peer-ratings  are 
available.  If  the  self-ratings  agreed  significantly  with  the  peer- 
ratings,  the  self-ratings  could  then  be  used  as  the  criterion  for 
validation  of  the  ET  and  CER  when  peer-ratings  were  not  obtainable 
in  sufficient  quantity.  In  addition,  if  the  supervisory  ratings  were 
significantly  correlated  with  the  peer-ratings,  then  the  supervisory 
ratings  could  be  used  as  the  criterion  for  validation  of  the  ET. 

The  intercorrelations  between  the  various  rating  methods  on  che 
"Overall  Performance"  scale  as  shown  in  Table  I  reveals  that  the 
supervisory  ratings  were  the  most  highly  related  to  the  peer-ratings 
having  a  correlation  coefficient  of  .3922.  Although  the  correlation 
coefficients  .2536  for  self-rating  vs  peer-ratings  and  .2463  for 
self-rating  vs  supervisory  rating,  respectively  were  somewhat  lower 
than  the  peer  vs  supervisor  correlation,  no  significant  differences 
were  found  between  the  correlation  coefficients  at  the  .05  signifi¬ 
cance  level.  However,  all  the  correlation  coefficients  were  signi¬ 
ficantly  different  from  zero  at  the  .01  level. 

The  intercorrelation  coefficients  between  the  rating  methods  on 
the  "Promotion  Qualification"  scale  are  shown  in  Table  II.  Again  the 
highest  correlation  (.4766)  was  between  the  peer-ratings  and  the  super¬ 
visory  ratings.  The  correlation  coefficients  were  .3111  and  .2397 
for  the  peer  vs  self  ratings  and  self  vs  supervisory  ratings,  re¬ 
spectively.  All  correlation  coefficients  were  significantly  different 
from  zero  at  the  .01  level.  The  peer  vs  supervisor  correlation  (.4766) 
was  significantly  different  from  the  self  vs  supervisor  correlation  at 
the  .05  level  (t  ■  2.4227).  No  significant  differences  were  found  be¬ 
tween  the  remaining  correlations. 

The  correlation  coefficients  presented  in  Table  III  show  that  the 
least  difference  between  the  two  rating  scales  existed  using  self-ratings 
with  a  correlation  of  .8489.  The  greatest  difference  (r  ■  .7296)  was 
found  with  the  peer-ratings.  Although  no  significant  differences  were 
found  between  these  coefficients,  it  appears  that  possibly  the  peer 
raters  make  somewhat  greater  discriminations  between  the  two  rating 
factors  than  do  the  self  or  supervisory  raters. 

Conclusion 

On  the  basis  of  this  study  it  appears  that  supervisory  ratings  would 
be  preferable  to  self-ratings  as  a  substitute  criterion  when  peer-ratings 
are  inaccessible.  However,  further  and  more  extensive  research  is  needed 
in  this  area  before  such  criterion  substitution  and  implementation  could 
be  incorporated  in  a  validation  program. 
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